fix: declare minimum versions for all core dependencies by dimitri-yatsenko · Pull Request #1653 · datajoint/datajoint-python

dimitri-yatsenko · 2026-11-05T13:33:42Z

Summary

Seven core dependencies had no version specifier (numpy, pandas, pyparsing, tqdm, networkx, pydot, packaging), and deepdiff had only a Python 3.15 cap. GitHub's dependency graph omits requirements without a specifier, so these packages were missing from the repository SBOM and from Dependabot alerts. This PR declares a tested minimum for each, and raises two existing minimums that did not work.

Package	Minimum	Reason
`numpy`	`1.22.4`	pandas 2.2's numpy minimum on Python < 3.11
`pandas`	`2.2.2`	First pandas built against numpy 2. pandas does not cap numpy, so pandas 1.5.3 with numpy 2 installs and then fails on import (`numpy.dtype size changed`); 2.2.1 still requires numpy < 2
`deepdiff`	`6.6.0`	`DeepHash(ignore_iterable_order=...)` appears in 6.4; before 6.6, deepdiff uses `np.float_`, which numpy 2 removed
`pymysql`	`1.0.0` (was `0.7.2`)	pymysql 0.x rejects the `ssl_disabled` connection option, so every MySQL connection fails on 0.7.2 through 0.10.1
`pyparsing`	`3.1.1`	`pp.DelimitedList` needs 3.1. 3.1.0 strips the quotes from quoted defaults, so `x = "" : varchar(…)` is declared with no default (see below)
`tqdm`	`4.66.3`	Excludes CVE-2024-34062
`networkx`	`3.0`	Tested minimum
`pydot`	`3.0.0`	pydot 2.x produces DOT that graphviz rejects for datajoint diagrams (`syntax error … near ','`)
`packaging`	`22.0`	Tested minimum

pixi.lock is refreshed; only the requirement metadata changes, no package versions.

Verification

Python 3.10, created with uv, every package above installed at exactly its minimum, other dependencies at current versions, full suite against MySQL, PostgreSQL, and SeaweedFS:

With these minimums (numpy 1.22.4): 1234 passed, 14 skipped. The test-only matplotlib was held at 3.8.4, since current matplotlib requires numpy ≥ 1.23.
Same, with numpy 2.2.6 and current matplotlib: 1234 passed, 14 skipped.
With pymysql 0.7.2 or 0.10.1: 542 errors, all Connection.__init__() got an unexpected keyword argument 'ssl_disabled'. 1.0.0, 1.0.2, 1.0.3, and 1.1.0 pass.
With pyparsing==3.1.0: the ~~ job tables and other tables with = "" defaults are declared NOT NULL with no default, and inserts fail with Field 'error_message' doesn't have a default value (MySQL) or NotNullViolation (PostgreSQL).
With pydot==2.0.0: 11 diagram tests fail; 3.0.0 through 3.0.2 pass.
DeepHash(..., ignore_iterable_order=False) checked directly on deepdiff 5.0 through 8.0: rejected as an unknown parameter before 6.4; fails with numpy 2 before 6.6.

Users with pyparsing 3.1.0, pydot 2.x, pymysql 0.x, or pandas < 2.2.2 installed get an upgrade from pip with this change.

numpy, pandas, pyparsing, tqdm, networkx, pydot, and packaging had no version specifier. GitHub's dependency graph omits such requirements, so they were missing from the SBOM and from Dependabot alerts. The minimums were verified by running the full test suite on Python 3.10 with each dependency at its minimum. pyparsing 3.1.0 and pydot 2.x fail it: pyparsing 3.1.0 strips the quotes from quoted defaults, and pydot 2.x produces DOT that graphviz rejects. tqdm>=4.66.3 excludes CVE-2024-34062.

MilagrosMarin

Thanks @dimitri-yatsenko! This is a useful cleanup, and the pyparsing and pydot findings are great catches.

1. pandas 1.5.3 + numpy 2 is allowed but fails on import. pandas 1.5.3 doesn't cap numpy, so pandas>=1.5.3 with numpy>=1.22.4 can resolve to pandas 1.5.3 + numpy 2.x. On Python 3.10, import pandas then fails:

ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The same pandas with numpy 1.26.4 imports fine. The minimum-versions run didn't hit this because numpy was pinned at 1.22.4 there. Suggest raising the bound to pandas>=2.2.2 (the first numpy 2–compatible release) or adding numpy<2 for older pandas.

2. deepdiff still has no lower bound. By the PR's own reasoning (unversioned requirements are left out of the SBOM and Dependabot), it would still be missing. Its only version marker is the <9 cap for Python 3.15.

Smaller:

I think the numpy reason ("oldest version supported on Python 3.10") is off, since numpy 1.21.x shipped 3.10 wheels. 1.22.4 matches pandas 2.1's numpy minimum, so it may be worth rewording once the pandas bound is settled.
pymysql>=0.7.2 wasn't in the minimum-versions run. Could be a follow-up.

Happy to re-review once updated!

- pandas>=2.2.2: pandas does not cap numpy, so pandas 1.5.3 with numpy 2 installs and then fails on import (numpy.dtype size changed). 2.2.2 is the first pandas built against numpy 2; 2.2.1 still requires numpy<2. - deepdiff>=6.6.0: DeepHash(ignore_iterable_order=...) first appears in 6.4, and deepdiff before 6.6 uses np.float_, which numpy 2 removed. - pymysql>=1.0.0: connections pass ssl_disabled, which pymysql 0.x does not accept, so every MySQL connection fails on 0.7.2 through 0.10.1. Full suite on Python 3.10 with every core dependency at its minimum passes, and again with numpy 2.2.6 in place of numpy 1.22.4.

dimitri-yatsenko · 2026-11-06T15:10:54Z

Thanks @MilagrosMarin, all four addressed in a98fd31.

pandas + numpy 2: Raised to pandas>=2.2.2. Confirmed on Python 3.10: pandas 1.5.3 with numpy 2 fails on import exactly as you describe, 2.2.1 still requires numpy<2, and 2.2.2 imports with numpy 2.2.6.
deepdiff: Now deepdiff>=6.6.0. DeepHash(ignore_iterable_order=...) is rejected before 6.4, and before 6.6 deepdiff uses np.float_, which fails with numpy 2. I checked that DeepHash detects changed values and reordering on 5.0 through 8.0.
numpy reason: You're right about 3.10 wheels. The table now says 1.22.4 is pandas 2.2's numpy minimum on Python < 3.11.
pymysql: Ran it. 0.7.2 and 0.10.1 fail on every MySQL connection (ssl_disabled is not accepted before 1.0), so the minimum is now pymysql>=1.0.0. 1.0.0 through 1.1.0 pass.

Full suite on Python 3.10 with every core dependency at its new minimum: 1234 passed, 14 skipped. Again with numpy 2.2.6: 1234 passed, 14 skipped. The PR description has the details.

MilagrosMarin

Thanks @dimitri-yatsenko, all four addressed in a98fd31 — and thanks for running the pymysql minimum rather than leaving it as a follow-up.

✅ pandas>=2.2.2 closes the pandas 1.5.3 + numpy 2 import failure
✅ deepdiff>=6.6.0 gives it a real lower bound, with the 3.15 cap kept
✅ numpy reason reworded
✅ pymysql>=1.0.0, matching the ssl_disabled option the MySQL adapter passes

The inline comments in pyproject.toml make each bound easy to revisit later. LGTM!

…nimums A new test job runs the full suite on Python 3.10 with every runtime dependency pinned at the minimum pyproject.toml declares. .github/scripts/minimum_requirements.py derives the pins and fails when a runtime dependency declares no minimum. deepdiff and pymysql now start at the first release without known vulnerabilities rather than the oldest that works: deepdiff>=8.6.2 (CVE-2025-58367, CVE-2026-33155; was 6.6.0) and pymysql>=1.1.1 (CVE-2024-36039; was 1.0.0). pixi.lock moves deepdiff from 8.6.1 to 9.1.0 where Python < 3.15.

dimitri-yatsenko · 2026-11-06T15:26:57Z

Adding two things on top of a98fd31 (cc4ca9e). @MilagrosMarin, could you re-review?

1. A CI job that keeps these minimums tested. test (minimum dependency versions, py3.10) runs the full suite on Python 3.10 with every runtime dependency pinned at its declared minimum. .github/scripts/minimum_requirements.py derives the pins from [project].dependencies. It fails when a runtime dependency has no minimum, so a bare numpy can't come back. It combines several lines for the same package (the 3.15 deepdiff<9 cap), takes the highest lower bound, normalizes names, and rejects wildcard versions and minimums that a requirement's own other version specifiers exclude. Optional extras are not pinned.

2. Minimums for deepdiff and pymysql raised past known vulnerabilities. Your review settled the oldest versions that work. Because datajoint now runs in secure environments, these two start instead at the first release without a published advisory:

	Oldest that works (`a98fd31`)	Now	Advisory
`deepdiff`	6.6.0	8.6.2	CVE-2025-58367 (class pollution in `Delta`, possible code execution), CVE-2026-33155 (memory exhaustion)
`pymysql`	1.0.0	1.1.1	CVE-2024-36039 (SQL injection via dict keys)

The comments in pyproject.toml keep your functional reasons and add the CVEs. pixi.lock moves deepdiff from 8.6.1 to 9.1.0 on Python < 3.15.

Verification. On Python 3.10, installed with pip exactly as the job does, at deepdiff==8.6.2 fsspec==2023.1.0 networkx==3.0 numpy==1.22.4 packaging==22.0 pandas==2.2.2 pydantic-settings==2.0.0 pydot==3.0.0 pymysql==1.1.1 pyparsing==3.1.1 tqdm==4.66.3, the full suite passes: 1234 passed, 14 skipped. A pip dry run confirms these are the versions the job installs. actionlint passes, and the branch merges cleanly with #1652 and #1654.

CVE-2025-58367 and CVE-2026-33155 are in deepdiff's Delta class and pickle_load. datajoint uses only DeepHash, so the oldest working version stays the minimum and Dependabot reports the advisories. pymysql stays at 1.1.1 for CVE-2024-36039.

dimitri-yatsenko · 2026-11-06T16:05:44Z

Done in a69650b, following the review on the advisory floors:

pymysql: (a), kept at >=1.1.1 for CVE-2024-36039.
deepdiff: (b), back to >=6.6.0, the oldest version that works. I read both advisories before deciding. CVE-2025-58367 is class pollution through the Delta constructor, leading to pickle deserialization. CVE-2026-33155 is unbounded allocation in _RestrictedUnpickler (pickle_load and Delta). datajoint uses only deepdiff.DeepHash (autopopulate.py:656-711 is the only use in src/), so neither applies. The comment in pyproject.toml says so, and Dependabot will still report the advisories. pixi.lock is back to deepdiff 8.6.1; the only lock change from a98fd31 is the pymysql requirement line.

On Python 3.10, installed with pip as the CI job does, at deepdiff==6.6.0 fsspec==2023.1.0 networkx==3.0 numpy==1.22.4 packaging==22.0 pandas==2.2.2 pydantic-settings==2.0.0 pydot==3.0.0 pymysql==1.1.1 pyparsing==3.1.1 tqdm==4.66.3, the full suite passes: 1234 passed, 14 skipped. Relative to the version you approved, the only floor change is now pymysql 1.0.0 → 1.1.1; the minimum-versions CI job is the other addition.

MilagrosMarin

Thanks @dimitri-yatsenko, the additions since a98fd31 look good:

✅ pymysql>=1.1.1 matches the CVE-2024-36039 advisory (fixed in 1.1.1)
✅ Keeping deepdiff>=6.6.0 makes sense: both advisories are in Delta/pickle_load, and DeepHash in autopopulate.py is the only use in src/
✅ The minimum-versions job is a nice guard. Failing on a dependency with no minimum keeps a bare requirement from coming back

Approving. Thanks for adding the CI coverage!

lum-agilyti

One thing: this overlaps with PR 1628, which moves pymysql out of the core dependencies into the mysql extra at >=0.7.2. Whichever merges second will need the 1.1.1 floor on the extra. Also, the description's table still lists pymysql at 1.0.0, while the diff is at >=1.1.1.

dimitri-yatsenko requested review from MilagrosMarin, lum-agilyti and ttngu207 November 5, 2026 13:33

MilagrosMarin requested changes Nov 6, 2026

View reviewed changes

dimitri-yatsenko mentioned this pull request Nov 6, 2026

ci: Dependabot watches Python dependencies, not only GitHub Actions #1652

Merged

dimitri-yatsenko requested a review from MilagrosMarin November 6, 2026 15:10

MilagrosMarin previously approved these changes Nov 6, 2026

View reviewed changes

dimitri-yatsenko mentioned this pull request Nov 6, 2026

ci: pin actions to commit SHAs and run workflows with read-only tokens #1654

Merged

dimitri-yatsenko dismissed MilagrosMarin’s stale review via c751171 November 6, 2026 15:26

dimitri-yatsenko force-pushed the fix/dependency-lower-bounds branch from c751171 to cc4ca9e Compare November 6, 2026 15:26

dimitri-yatsenko requested a review from MilagrosMarin November 6, 2026 15:26

MilagrosMarin approved these changes Nov 6, 2026

View reviewed changes

dimitri-yatsenko modified the milestones: v2.3.6, v2.3.7 Nov 6, 2026

dimitri-yatsenko mentioned this pull request Nov 6, 2026

fix: dj.Top is honored when a limited expression is used in another query #1657

Merged

lum-agilyti reviewed Nov 10, 2026

View reviewed changes

dimitri-yatsenko mentioned this pull request Nov 10, 2026

fix: aggr, extend and union keep unmatched rows when the right operand is restricted #1655

Merged

dimitri-yatsenko merged commit 32ed562 into master Nov 10, 2026

dimitri-yatsenko deleted the fix/dependency-lower-bounds branch November 10, 2026 19:52

Navigation Menu

Repository navigation

fix: declare minimum versions for all core dependencies - #1653

fix: declare minimum versions for all core dependencies#1653
dimitri-yatsenko merged 4 commits into
masterfrom
fix/dependency-lower-bounds

dimitri-yatsenko commented Nov 5, 2026 •

edited

Loading

Uh oh!

MilagrosMarin left a comment

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

MilagrosMarin left a comment

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

MilagrosMarin left a comment

Uh oh!

lum-agilyti left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dimitri-yatsenko commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Verification

Related

Uh oh!

MilagrosMarin left a comment

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

MilagrosMarin left a comment

Choose a reason for hiding this comment

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

dimitri-yatsenko commented Nov 6, 2026

Uh oh!

MilagrosMarin left a comment

Choose a reason for hiding this comment

Uh oh!

lum-agilyti left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dimitri-yatsenko commented Nov 5, 2026 •

edited

Loading